Omnidirectional imaging optics with 360°-seamless telescopic resolution

ABSTRACT

A multifacet mirror comprises a catoptric structure configured to reflect light from a first field of view onto an image plane comprising a plurality of sensors, the catoptric structure having a surface comprising a plurality of facets separated by a plurality of catoptric regions; wherein a facet of the plurality of the facets has a second field of view that is smaller than the predetermined field of view; and wherein a catoptric region of the plurality or catoptric regions between two facets of the plurality of facets is configured to reflect light from a world point within the first field of view onto two of the sensors of the plurality of sensors.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/311,880, filed Mar. 9, 2010.

TECHNICAL FIELD

The present invention relates generally to optics, and moreparticularly, some embodiments relate omnidirectional imaging.

DESCRIPTION OF THE RELATED ART

The ultimate goal of omnidirectional imaging optics is to realize360°—seamless telescopic resolution; i.e., with less than 1 mrad,equivalent, for example, to resolution of 1 m per 1 km distance, withinsome range of elevation angles, where 360°—means full azimuthal angle.Prior omnidirectional imaging systems only partially realize this goalby directing a telescopic camera into specific narrow FOV (field ofview) regions. By moving the camera, high resolution may be achieved ata given azimuthal angle, but not, at the same time for all angles,limiting such solutions to a single target at a given time. Increasingthe number of such high-resolution cameras, or even placing them closeto each other creates obvious seams at the boundary (interface) betweentwo adjacent cameras. This, in turn, creates image discontinuity when atarget from a FOV of one camera into the other one.

Furthermore, the pan/tilt/zoom (PTZ) operations usually require threeseparate bulky mechanical actuators—two for controlling camera viewingdirection (pan and tilt) and one to vary the effective focal length ofthe camera lens. Mechanical actuators, especially for pan and tilt,require periodical calibration and alignment for proper angular imageregistration. Another drawback of such a system, besides the need formechanical actuators, is that such a camera can only observe a scene ina small solid angle, rather than the entire 2π solid angle (entirehemisphere).

BRIEF SUMMARY OF EMBODIMENTS OF THE INVENTION

According to various embodiments of the invention a catadioptric systemis presented comprising a multi-facet mirror, a sensor array, and imageprocessing modules. The multi-facet mirror is configured to image worldpoints from a field of view such that world point rays are not lost inphysical spaces between sensors of the sensor array. Each facet of themirror has an associated sensor in the sensor array. The facets areseparated by catoptric regions, “seams,” that are configured to reflectrays from world points that would otherwise be lost in the physical gapsbetween sensors onto edge regions of the arrays surrounding the gap. Theimages created by the sensor arrays therefore include overlappingregions that allows the image processing modules to stitch the imagestogether into a composite image by aligning the overlapping regions.

According to an embodiment of the invention, a multi-facet mirrorcomprises a catoptric structure configured to reflect light from a firstfield of view onto an image plane comprising a plurality of sensors, thecatoptric structure having a surface comprising a plurality of facetsseparated by a plurality of catoptric regions; wherein a facet of theplurality of the facet has a second field of view that is smaller thanthe predetermined field of view; and wherein a catoptric region of theplurality of catoptric regions between two facets of the plurality offacets is configured to reflect light from a world point within thefirst field of view onto two of the sensors of the plurality of sensors.

Other features and aspects of the invention will become apparent fromthe following detailed description, taken in conjunction with theaccompanying drawings, which illustrate, by way of example, the featuresin accordance with embodiments of the invention. The summary is notintended to limit the scope of the invention, which is defined solely bythe claims attached hereto.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The drawings are provided for purposes of illustration only andmerely depict typical or example embodiments of the invention. Thesedrawings are provided to facilitate the reader's understanding of theinvention and shall not be considered limiting of the breadth, scope, orapplicability of the invention. It should be noted that for clarity andease of illustration these drawings are not necessarily made to scale.

Some of the figures included herein illustrate various embodiments ofthe invention from different viewing angles. Although the accompanyingdescriptive text may refer to such views as “top,” “bottom,”“horizontal,” or “side” views, such references are merely descriptiveand do not imply or require that the invention be implemented or used ina particular spatial orientation unless explicitly stated otherwise.

FIG. 1 illustrates a catadioptric system implemented in accordance withan embodiment of the invention.

FIG. 2 illustrates an analog and digital imaging situation.

FIG. 3 illustrates a sensor array and non-recorded image portions from asingle faceted catadioptric system.

FIG. 4 illustrates a view of a cylindrical catoptric mirror illustratingthe dead zones from a dead angle

FIG. 5 illustrates a catoptric structure with reduced or eliminatednon-imaged zones implemented in accordance with an embodiment of theinvention.

FIG. 6 illustrates a multifacet mirror system implemented in accordancewith an embodiment of the invention.

FIG. 7 illustrates a dual facet mirror implemented in accordance with anembodiment of the invention.

FIG. 8 illustrates a multifacet mirror system implemented in accordancewith an embodiment of the invention.

FIG. 9 illustrates a multifacet mirror system with azimuthal andhorizontal facets implemented in accordance with an embodiment of theinvention.

FIG. 10 illustrates a sensor array corresponding to half of themulti-facet mirror of FIG. 9.

FIG. 11 illustrates a zoom-in operation implemented in accordance withan embodiment of the invention

FIG. 13 illustrates a zoom-out operation implemented in accordance withan embodiment of the invention.

FIG. 13 illustrates angular resolution differences resulting from somemirror geometries.

FIG. 14 illustrates the relationship between resolution and height in acatadioptric system.

FIG. 15 illustrates a multi-zone azimuthal resolution deploymentimplemented in accordance with an embodiment of the invention.

FIG. 16 illustrates a multi-zone digital display system implemented inaccordance with an embodiment of the invention.

FIG. 17 illustrates a catadioptric geometry for a panoramic 180° viewsystem implemented in accordance with an embodiment of the invention.

FIG. 18 illustrates a display screen field of view implemented inaccordance with an embodiment of the invention.

FIG. 19 illustrates an example computing module that may be used inimplementing various features of embodiments of the invention.

The figures are not intended to be exhaustive or to limit the inventionto the precise form disclosed. It should be understood that theinvention can be practiced with modification and alteration, and thatthe invention be limited only by the claims and the equivalents thereof.

DETAILED DESCRIPTION OF THE EMBODIMENTS OF THE INVENTION

The present invention provides systems with seamless pixels andelectronic zoom, called EL ZOOM, and electronic PTZ (or EL PTZ) whichreplace prior art mechanical (with motion) and electro-opticalequivalents. The EL-PTZ does not necessarily have any mechanical parts(no motion), nor electro-optical parts, or acousto-optic parts. It isbased on opto-electronic integration of well-established pixel arrays(PAs) similar to those used for present imaging optical cameras, intosuperpixel arrays (SPAs) that have obvious mechanical seams, but thatare seamless optically; i.e., integrated omnidirectional image isseamless. This seamless property of the SPAS is due to a multifacetoptical reflective component. The system achieves a high (telescopic)resolution as well as efficient electronic image stabilization withoutimage actuators.

Analog angular image resolution is estimated from the Raleigh criterion,which is based on Airy (intensity) patterns, in the form:

$\begin{matrix}{y = \left\lbrack \frac{2{J_{1}(x)}}{x} \right\rbrack^{2}} & (1)\end{matrix}$where J₁(x) is the Bessel function of the 1st kind and 1st order, withx:x=πd(Δθ)/λ  (2)where (Δθ) is angular distance from the center, d is eyepiece (orprojection lens) diameter, and λ is optical wavelength (in air). Thespatial distance from the center of the Airy pattern, δ, is from Eq.(2):

$\begin{matrix}{\delta = {{f({\Delta\theta})} = {\frac{x\;\lambda\; f}{\pi\; d} = {\frac{x\;\lambda}{\pi}f\#}}}} & (3)\end{matrix}$where f is eyepiece focal length, and f#=f/d. The Airy pattern,described by Eq. (1), is a diffraction pattern with adjacent maxima andminima (rings), defined by the Bessel function, summarized in Table 1,for λ=0.5 μm, and f#=2.5.

TABLE 1 Maxima and Minima of Airy Pattern (Analog Resolution), for λ =0.5 μm and f# = 2.5 Maximum/Minimum x δ y 1st maximum 0 0 1 1st minimum(1^(st) ring) 1.22 π 1.52 μm 0 2nd maximum 1.635 π 2 μm 0.0175 2ndminimum(2^(nd) ring) 2.233 π 2.79 μm 0

In digital imaging systems, resolution is impacted by the Nyquistcriterion, which requires a sampling interval at least twice as high asthe highest spatial frequency to be accurately preserved in the digitalimage. Assuming, for illustration, that the size of each pixel is 4 μmand the space separating the pixels is 1 μm, (thus, the pitch is 5 μm),then the Nyquist resolution is 10 μm or δd_((R))=10 μm.

FIG. 1 illustrates a general catadioptric system. A convex mirror 40reflects incoming light rays 41 onto a camera 43, for example through apinhole 44 (i.e., in a camera obscura). As used herein, the term “pole”will refer to the point of the mirror closest to the focal region 44,and the ter “equator” refer to the circumference of the x-y crosssection furthest from the pole (circumference 47 in the Figure), wherethe z direction is the direction normal to the focal plane.

The single-viewpoint criterion is the requirement that a catadioptricsensor only measure the intensity of light passing through a singlepoint in 3-D space. This is equivalent to the fact that each world point42, within δw-vicinity, and dν-infinitesimal solid angle, and dφ—itslinear angle equivalent, has one chief ray 41, that, after reflectionfrom the surface of mirror 40, passes the pinhole 44. Then, dν-angletransforms into do angle, and then to dA-pixel area in camera 43. If themirror 40 is hyperboloid with a first foci 45 and a second foci atpinhole 44, then the world lines 41 are those lines directed at thefirst foci 45. In a real catadioptric system, the pinhole is replaced bya lens and aperture. In this case, the chief ray is transformed into abundle of rays from any world point, resulting in some blur.

To achieve 360° azimuthal (horizontal) view, the mirror 40 represents ashape that is swept through a full 360° rotation, such as a hyperboloid,or other rotationally symmetric shape. A vertical field of view 46 isdefined by the camera placement and shape of the mirror 40.

FIG. 2 illustrates a general digital imaging situation. In the figure, acircle 81 with radius r will be in aged by a pixel 83 array 84. Theresolution of the image 82 is determined by the Nyquist criterion, whichrequires that the number of resolving elements 80 2πr/δr should be halfthe number of equivalent pixels 83, 2πN_(r), where N_(r) is thehalf-size of the equivalent pixel array 84.

$\begin{matrix}{{\frac{2\pi\; r}{\delta\; r} = \frac{2\pi\; N_{r}}{2}};\mspace{14mu}{N_{r} = \frac{2r}{\delta\; r}}} & (4)\end{matrix}$The resolving power of the imaging system, P_(R), is defined as theinverse of the system resolution in meters/km, such as 1 m/1 km, or 1mrad; thus

$\begin{matrix}{\frac{1}{P_{R}} = \frac{\delta_{r}}{r}} & (5)\end{matrix}$so, from Eq. (4):N_(r)=2P_(R)  (6)

FIG. 3 illustrates a system for increasing the resolving power of acatadioptric image. A catadioptric imaging system having a 360°azimuthal FOV forms an annular image 120 on the sensor plane. Thedistance 121 from the inner radius to the outer radius of the annulus isdetermined by the vertical FOV and mirror used. In the case of less than360° azimuthal FOV the image formed will be a section of the annulus.Available digital image detectors, such as CCD or CMOS based detectors,“pixel arrays”, have limited resolutions. In the illustrated embodiment,a plurality of pixel 122 arrays 126, 127, 128, and 129 are arranged in asuper-array. For example, four quadratic 16-megapixel pixel arrays canbe used to obtain a 64-megapixel super-array. If, in general, thehalf-size of a number of pixels for PA, is N_(r), as in Eq. (6), then,this number increases twice, to N_(x), to obtain:N_(x)=4P_(R)  (7)This relation is summarized in Table 2, for various resolving powers.

TABLE 2 Pixel Resolution v.s. Resolving Power 1/P_(R) 1 m/1 km 1 m/2 km1 m/3 km 1 m/5 km 1 m/10 km 1/P_(R) 1 mrad 0.5 mrad 0.3 mrad 0.2 mrad0.1 mrad (radians) P_(R) 1,000 2,000 3,000 5,000 10,000 N_(x) 4,0008,000 12,000 20,000 40,000 Megapixels 16 64 144 400 1,600

However, when tiling multiple image detectors, physical gaps 123, 124,125, and 130 will exist between the sensors. In the case of an annularimage 120 reflected off of a simple surface of revolution, such as ahyperboloid the portions of the image 120 that exist at the gap regionsare unable to be imaged. Accordingly, the rays from the omni-directionalworld points corresponding to those regions are lost. This results innon-imaged zones within a “dead angle” 131, 2φ. FIG. 4 illustrates aview of a cylindrical catoptric mirror illustrating the dead zones fromthe dead angle 131. As the figure illustrates, the effects of the deadzone increase in the vertical field in the direction towards the base ofthe mirror.

FIG. 5 illustrates a catoptric structure with reduced or eliminatednon-imaged zones. FIG. 5A illustrates a top-down view of the resultingcatoptric mirror, while FIG. 5B illustrates a side view of the catoptricmirror. The mirror comprises a plurality of facets 154, 155, 156, and157. In one embodiment, the facets are connected by flat contours 162,165, 164, and 163, and flat region 166. Each facet has a center ofcurvature, 158, 160, 161, and 159, respectively that is displaced fromthe center 149 of the mirror's circumference. Each facet then comprisesa surface of revolution of a conic section or a skewed conic section,with the axis of rotation at the center of curvature. For example, insome embodiments each facet comprises a section of a paraboloid or ahyperboloid. To form the catoptric multi-facet mirror, in someembodiments, a plurality of sections of a paraboloids or hyperboloidsare formed, for example, 95° sections of the surfaces of rotation. Thesections are then joined together to form the multi-facet mirror havingredundant seams between the facets.

In one embodiment, the displacement is such that the centers ofcurvatures correspond to the corners of the pixel arrays 150, 153, 152,and 151 on the image plane. Accordingly, in the illustrated embodimentall omni-direction chief rays are accepted by the mirror and mapped to apixel array. Accordingly, no chief rays are lost in the regions betweenthe pixel arrays. In some embodiments, the single viewpoint criterion ismaintained by modifying the x-z profiles 167 and 168 of the mirror. Inother embodiments, the single viewpoint criterion is maintained for eachfacet of the mirror such that each corresponding pixel array has asingle viewpoint.

With flat contours 162, 165, 164, and 163 between the facets, the pixelarray dimensions and spaces between the pixels and the facet dimensionsshould be precisely controlled, so that each facet of the mirror mapsaccurately to its corresponding pixel array. Otherwise, discontinuitiesmay occur or information from some world points may be lost. FIG. 6illustrates a multifacet mirror system that alleviates theserestrictions. FIG. 6A illustrates the mirror in relation to the imagesensor; FIG. 6B illustrates a plurality of x-y cross sections of themirror; and FIGS. 6C and 6D illustrate two perspective views of themirror. The catoptric mirror comprises a plurality of facets, 200, 201,202, and 203 as before with respect to FIG. 5. Each facet has a focalpoint corresponding to a corner of a different pixel array, forming alocus 206 of single view points on the focal plane. However, in thisembodiment, the facets are joined in with concave seams 204, such as acusp 211 with curvatures on either side of the seam that result inredundant or overlapping on the edges of the pixel arrays, such asregions 208 and 207 on pixel arrays 209 and 210. For example, in oneembodiment, each facet of the mirror has a horizontal field of viewbetween about 100° and 105°. Because of the seams 204, within an angularregion determined by the dimensions of the seams 204, rays from worldpoints are mapped to two separate pixel arrays. For example, in seam204, rays 212 from world points 235 are imaged by pixel array 209 withinregion 208 and rays 213 are imaged by pixel array 210 within region 207.The redundant images created by concave contours of seams 204 are thenused to stitch together the images from each pixel array in electronicimage processing.

In further embodiments, the number of facets in the catoptric mirror maybe varied. For example, FIG. 7 illustrates a two facet catoptric mirrorand its resulting image geometry. Facets 251 and 252 are separated by aconcave seam 250. Accordingly, each of the facets has a higher than 180°FOV, resulting in regions of overlap 255 and 256 in the images 254 and233 on the sensor plane. Each pixel array 258 and 257 therefore has aoverlapping redundant area that allows the images to be stitchedtogether in electronic image processing without losing rays in the gap259 between the sensors. FIG. 8 illustrates a 12 facet catoptric mirror281 and a corresponding array of 12 image sensors 280. If for example,each of the image sensors 280 had a resolution of 1,080×1,920,corresponding to commercially available HDTV camcorder sensors, then thecatadioptric system would have an ultimate resolution of about12,980×1,920 pixels.

In still further embodiments, multifacet catoptric mirrors may havefacets that are separated azimuthally as well as circumferentially. FIG.9 illustrates one such multifacet mirror. The facets of the mirror aredivided along the circumference of the x-y plane using concave seams.For example, facets 301 and 300 are separated by seam 307. Facets arefurther divided by similar concave seams in the azimuthal x-z, and y-zplanes. For example, facets 300 and 302 are separated by concave seam308. In some embodiments, the azimuthal seams 308 may be formed byincreasing the curvature of the mirror in the region near the seam. Forexample, if a cross section of a facet of the mirror was a parabola withan equation y=3*x^2, then the azimuthal seam 308 might compriseparabolic regions with an equation y=7*x^2.

A separate sensor, such as a CCD or CMOS pixel array, is provided foreach facet. The seams separating the facets provided overlapping imagesin regions on the edges of the sensors, allowing image stitching andpreventing loss of rays from world points corresponding to the physicalgaps between the sensors. This arrangement can be used to reduce therequirements for high-resolution sensors, for example as in theembodiment of FIG. 8.

The use of azimuthal facets also allows the use of the differentresolution sensors corresponding to different zones within the field ofview. FIG. 10 illustrates a sensor array corresponding to half of themulti-facet mirror of FIG. 9. In these embodiments, the multifacetmirror is divided into three azimuthal zones. An inner zone of facets,including facet 304, a middle zone of facets, including 302, 303, and306, and an outer zone of facets, including 300, 301, and 305. Thesensor array 350 is divided into corresponding zones of sensors ofvarying resolution. An inner zone of sensors, including sensors 358,357, and 359, corresponds to the inner zone of facets. A middle zone ofsensors, including sensors 354, 355, and 356, corresponds to the middlezone of facets. An outer zone of sensors, including sensors 351, 352,and 353, corresponds to the outer zone of facets. In some embodiments,the sensors in the different zones may have different resolutions. Forexample, the sensors in the outer zone may have twice the resolution ofthe sensors in the middle zone, which in turn have twice the resolutionof the sensors in the inner zone. In some implementation, objects ofinterest will tend to be within a relatively small angle to the horizon.For example, all objects up to 270 m in height that are 1000 m away fromthe mirror will be within 15° of the equator of the mirror. The use ofgreater resolution sensors can compensate for the loss of acuity thatwould otherwise be present from the conversion of angular resolution toCartesian resolution.

Once an image of sufficiently high resolution is produced using asuper-pixel array, mechanical and electro-optic zooming can be avoidedwithout the use of electronic interpolation zooming. In some embodiment,an all electronic zoom is implemented based on spatial sampling rateswitching. FIG. 11 illustrates a zoom-in operation tented in accordancewith an embodiment of the invention. After image stitching, an image 400comprises an array of pixel values 402. Because of the high resolutionobtained from the super-pixel array, the resolution of the image 400 ishigher than what can be used by image monitor 403. In a zoom-inoperation, a region 401 is selected from the image 400 and displayed onthe image monitor 403. Without interpolation, a maximum resolutionoccurs when each pixel 402 of the image within region 401 is displayedby a pixel 404 of the image monitor. Pan and tilt operations may then beperformed by translating the selected area 401 within the image 401.

FIG. 12 illustrates a zoom-out operation implemented in accordance withan embodiment of the invention. Here, to display more of the image 400on the image monitor 403, the pixel values 402 from a region of pixels401 is correlated and displayed on a single pixel 404 of the imagemonitor 403. Zooming in and out on the image 400 may then be performedby correlating larger and small numbers of pixels 402 of the image anddisplaying the correlated regions on the image display 403.

In catadioptric imaging, constant angular resolution translates into aneed for greater horizontal resolution near the outer circumference ofthe annular image when compared to the inner circumference. FIG. 13illustrates this problem. In FIG. 13 the annular image resulting from anomni-directional imaging system is divided into 6 regions 475, each 60°.If the inner region 478 halfway between the inner radius and the outerradius, is to be mapped to a rectangular screen with 640 pixels per lineof video, the inner circumference 476 must be imaged with 210 pixels perline per 60° and the outer circus reference 480 must be imaged with 712pixels per line per 60°. To achieve this, in some embodiments, a arrayof sensors corresponding to the needed outer resolution may be used. Forexample, here 6 image sensors may be arranged in a radial array, eachmore than 712 pixels×480 pixels (assuming a 640×480 image or video), asillustrated with respect to FIG. 8. In this embodiment, a 6 facet mirrormay be used with the 6 facets arranged circumferentially. In order toallow for overlapping regions, in some embodiments, each of the 6 facetsmay be separated by seams configured to allow each facet to imageslightly more than 60°, such as 63°. Alternatively, a multifacet mirrorwith facets arranged both circumferentially and azimuthally may be usedto allow use of a multi-resolution sensor array, as discussed withrespect to FIG. 9.

In addition to developing a sensor array with sufficient resolution,generally the image will be displayed in a rectangular, Cartesian grid.A variety of transforms from the polar annular image to a Cartesiandisplay may be used. For example, in one embodiment, a surface transformis applied. Here, the annular image is mapped onto a surface, which istransformed directly onto the Cartesian grid without the need forpixel-by-pixel transforms.

Generally, a 1360×1360 pixel omnidirectional image will exhibit qualitysimilar to that of a standard definition image from a 60° field of viewlens. The unwrapped image resolution will be about 3864×480 pixels. Inorder to produce electronic zoom the number of pixels must be an integermultiple of the base 1360×1360 resolution, as shown in Table 3.

TABLE 3 Dependence of Imaging Sensor Resolution on Zoom Factor Zoom 1×2× 4× 6× 8× 10× Resolution 1360 × 2720 × 5440 × 8160 × 10880 × 13600 ×1360 2720 5440 8160 10880 13660

In further embodiments, all electronic frame stabilization can beimplemented in image processing. Because the catadioptric system withmultiple sensor arrays and with electronically implemented pan, tilt,and zoom, constitutes single rigid body, without moving parts, itsmotion can be electronically synchronized, with some fiducial point; orstable reference point, such as a star, or some static point at theground. This may be implemented when the imaging system is installed ona moving platform, such as an avionic platform like a UAV (UnmannedAerial Vehicle). In such a case, the catadioptric system is calibratedwith a reference point that is static in respect to platform. Thereference point is a specific fiducial world point and results in afiducial pixel (FP) being an image of this point. If this image isdeviating with respect to the FP, this deviation may be correctedelectronically by electronically translating the viewing region, asdescribed with respect to FIGS. 11 and 12. The catadioptric systemsdescribed herein can also work as a hyperspectral imager, or multispectral imager, by applying expanded from visible into infrared, or UV.

Eqs. (4), (5), and (6) for the Nyquist criterion of catadioptric system,describe horizontal (azimuthal) digital resolution. For the vertical(elevation) digital resolution, equivalent of Eq. (6) has the form:

$\begin{matrix}{{\frac{({FOV})r}{\delta\; r^{(v)}} = \frac{({FOV})\Delta\; N_{r}}{2}};\mspace{14mu}{{\Delta\; N_{r}} = \frac{2r}{\delta\; r^{(v)}}}} & (8)\end{matrix}$where the (FOV) is elevation (FOV), such as (+70°, −20°), or FOV=90°,δr^((v))—is vertical (elevation) resolving element, and ΔN_(r) isdiagonal number of pixels, equivalent to this FOV. The equivalents ofEqs. (5) and (6) have the form:

$\begin{matrix}{\frac{1}{P_{R}^{(v)}} = \frac{\delta\; r^{(v)}}{r}} & (9) \\{and} & \; \\{{\Delta\; N_{r}} = {2P_{R}^{(v)}}} & (10)\end{matrix}$where P_(R) ^((v))=1000 is equivalent to vertical resolution of 1 mrad,or for example 1 m per 1 km (vertical).

In both horizontal and vertical resolutions, the resolving power, eitherP_(R) (for, horizontal resolution), or P_(R) ^((v)) (for verticalresolution) as in Eqs. (6) or (10), respectively, is defined byequivalent number of pixels, N_(r), or ΔN_(r). Of course, higher numberof pixels, higher resolving power, in both horizontal and verticalcases. To achieve very high resolution, such as 2.5 cm resolution at 1km distance, of 0.025 mrad, equivalent to resolving power of 40,000, inultrawide field of view (FOV) of 360°×45°, or half-view of 180°×45°(tunable up to 90°), the resulting sensor, or the SPA, should have125,000 pixels per line and a total of 30,000 semiannual lines. Toachieve this array size, a larger number of low-resolution sensors, orPAs, can be multiplexed together. The physical gaps between thelow-resolution sensor are obviated by the multifacet catoptric mirrorsused in the catadioptric system, as described above. Such highresolution in the sensor device will provide very high detail without aneed to mechanically change optical system focal length, as offered byprior mechanical zoom operations.

The multi-facet mirrors, for example as described with respect to FIGS.5-9, are reflection type optical elements that compensate for theboundary effect by changing angular distribution of the reflectedoptical rays. In general, the reflective 360° optical light collectingdevice is based on a multi-facet mirror. This smooth surface must bemodified to compensate the gaps between sensor chips (or, PAs), bydividing the mirror surface into sections, as shown in the Figures. Eachsection covers slightly more than an equal division of the FOV. Forexample, a mirror with 4 facets arranged circumferentially might haveeach section (facet) covering ˜93° FOV. This slight ˜3° overlapcompensates for the physical gap between the sensors. The final overlapangle will depend on the chip boundary thickness. In furtherembodiments, further modifications are used, such a some skew shift ofrays in order to satisfy the single view point criterion of acatadioptric system. In other embodiments, the circular symmetry of theprojection lens (eyepiece) is modified into multi-viewpoint topology ofthe catadioptric system. In some embodiments, the multi-viewpointtopology provides a viewpoint for each sensor of the sensor array.Digital image stitching and modification allows assembling the multipleimages having multiple viewpoints into a single combined image.

The requirement for 40,000×80,000 resolution of the SPA, or super-pixelsensor array can be reduced based on the analysis of a 187°×45° FOV as afunction of distance and elevation angle. As the distance increases, theelevation angle decreases at a constant resolving detail of 2.5 cm asshown in FIG. 14. The increased resolution near the horizon may be usedin some implementations because more detailed imaging of sites near thehorizon may be necessary. For example, physical object localization, forexample, through trees or in buildings, typically requires higherresolutions near the horizon. With distance, even tall buildings arelocated at lower elevations. To image objects up to 250 m tall, a 45°elevation angle is needed for object 250 m away, 30° at 500 m, and to15° at 1000 m. Even at 1000 m, the height of objects covered at 2 cmresolution is 270 m, sufficient for urban terrain. The importance ofthis remark is that the resolution requirements can be reduced bydividing the sensor area into three semi-circles (zones) 480, 481, and482 with different pixel densities as shown in FIG. 15. This azimuthalzone concept is discussed further with respect to FIGS. 9 and 10.

The non-uniform pixel density solution, and the zone concept, ingeneral, can be applied to specific operative requirements, such as thesniper localization. Other operative requirements, such as: perioscopicvision, border control, and various video surveillance scenarios candictate different zone distributions, and different pixel densitydistribution. In any case, the multi-facet zone mirrors can reduce thetotal number of pixels required, while preserving required resolution.

In summary, the zone mirror concept, generalizes the multi-facet mirrorarchitecture from horizontal facets into both horizontal and verticalfacets. It allows, in general, to further expand the total number of SPApixels, to further increase resolution, both horizontal (azimuthal) andvertical (elevation) one. It allows, also, to differentiate pixeldensities, according to the specific requirements in order to maximizesystem resolution.

In addition to spatial-seamless SPA/multi-facet firmware, someembodiments employ software based on temporally-spatially-seamlessalgorithm, eliminating temporal (and spatial) seams, by improvedunwrapping and stitching. In particular, in the case of real-time videounwrapping, a standard transformation of circular polar coordinates toCartezian coordinates at 30 frames per second (“fps”), is used. Previouspixel based transformations requires point-to-point interpolation andthus, distort results. In some embodiments, surface-based imagetransformation are employed. Accordingly, resulting image will be onlyminimally distorted, because of this advanced transformation technology,applicable have to catadioptric systems that apply quasi-parabolicmirrors. In the multi-sectional (multi-facet) case, this transformationis applied to each quadrant and stitched, as explained below.

Conventional stitching algorithms for panoramic (catadioptric) imagingrequire prior information, and often require iterative search. Theseprior stitching methods cannot operate in real time (or, 30 fps),especially in high-resolution video with limited prior information. Someembodiments employ a suboptimal but effective stitching algorithm thatwill adaptively perform solution searches for video manipulation bymeans of temporal and progressive processing. Convergence time isevaluated to optimize this approach to meet resolution and displaytiming requirements. The process is based on gradient descentalgorithms, which looks for a solution based on so-called cost functionsto adjust (spatial) image scams with minimum visible transitions.

The basic principle of the proposal sea less approach is to minimize acost function defined by a measure of dissimilarity between thederivations of the stitched image and the input images. Instead of aglobal evaluation for the derivatives, as in case of prior art approach,we provide this evaluation only for local overlapping regions, or otherpredefined regions, to evaluate the derivations and find a solution. Theproposed method is only suboptimal for single-frame estimation, but thespatial domain evaluation is an integral part of temporal evaluation,which additionally improves the process of minimizing, cost function.This method is effective under assumption that the video stream does notchange dramatically from frame to frame (within a single scene), andthat the variations are smooth. Changes of image characteristics areadaptively followed with weighted correlations according to the specifictimeline, meaning that current data weights more heavily than the pastdata. The optimization procedure is as follows: In order to estimatestitched image (I), Î, the cost function, C_(d), is minimized withrespect to two input images, I₁, I₂, in the form:C _(d)(I;I ₁ ,I ₂ ,w)=d(∇Î,∇I ₁ ,t ₁ ∪w,w)+d(∇Î,∇I ₂ ,t ₂ ∪w,U−w),  (11)where w is a weighting value, T is the area in image, ω-is theoverlapped area, ν-symbol means “within,” and d is the distance betweena and b, on c:

$\begin{matrix}{{d\left( {a,b,c,w} \right)} = {\sum\limits_{q \in c}{{w(q)}{{{{a(q)} - {b(q)}}}_{p}.}}}} & (12)\end{matrix}$

The dissimilarity cost function, C_(d), is defined by the distancebetween derivatives. The advantages of this method is that thedissimilarity function, C_(d), invariant to the overall (absolute)intensity of an image, and it is relatively insensitive to globaldifference between input images, if they are smooth.

The panoramic video display requires high resolution, typically6432×1024 pixel resolution for a 4 megapixel omni-directional ages(2048×2048). Because of the limitations of display devices, includingCRTs, LCDs, and projectors, panoramic imagery cannot be displayed at itsnative resolution, and thus is presented with less than the originalimage quality. Two levels of digital zoom are implemented; one losslesszoom and the other lossy. Lossless zooming manipulates and displays theimage with quality less than or equal to that of the original imagewithout interpolation (stretching). Lossy zooming interpolates theoriginal image to an expanded image that is bigger than the zoom area'snative resolution.

For motion stabilization purposes, in order to eliminate unwanted themotion of the camera, some embodiments compute the affine transform, awell-known transformation, that maps each frame into a given referenceframe, selected from a sequence of frames corresponding to approximatelyone second of video. It is assumed that the video scene consists ofstatic objects and moving objects, with the moving objects occupyingsignificantly fewer pixels than the static fixed objects, from whichcamera motion is derived. The affine transformation matrix includestransformation from the reference point (x,y) into the current frame(x′,y′), in the form of orthogonal matrix, representing rotation angleθ, and scale factor, s, plus translation vector: (t_(x), t_(y)). Wecompensate only unwanted motion, while the slow and smooth intentionalmotion (zooming, penning, natural direction change) is preserved. Theunwanted motion is abrupt, parasitic camera movements. These movementscan be separated, intentional and un-wanted, by a low-pass filtering inthe time domain, such as Kalman filtering.

As described above, the entire panoramic FOV, up to 360° is recorded atonce. Regions of interest are then selected electronically by software.Proper pixel clustering in the image device allows for a plurality ofresolution levels. FIG. 16 illustrates this situation. At the highestresolution level 551 each pixel 550 is displayed on a separate pixel ofthe screen 552. As illustrated, this allows only a reduced region of thetotal image to be displayed at a given time. At the middle resolutionlevel 553, clusters of pixels 554 are correlated into pixel values.These correlated pixel values are then displayed on screen 552. Thisallows a larger, but still reduced, region of the total image to bedisplayed. At the lowest resolution level 556, the clusters from themiddle resolution 554 are further correlated to form clusters 555. Theseclusters are then displayed on the screen 552. For example, if the wholeimage has 27,000 horizontal pixels, then the middle resolution level maycorrelated 3 pixels (9 total when vertical pixels are included) percorrelated pixel value to generate the values for level 553. Resultingin 9,000 effective horizontal pixels for the middle resolution level553. Another round of correlating 3 pixels, results in a 3,000 effectivehorizontal pixels for the low resolution level 556.

The minimum number of superpixels, defining horizontal and verticaldigital resolution, may be derived based on so-called Johnson resolutioncriterion rather than Nyquist criterion. The Johnson criterion definesthe minimum number of linear pixels per target (an object of interest),for: detection (D), recognition (R) and identification (I), as: 5, 10,and 15, respectively. The resolving elements, δl, are calculated inTable 4 for vehicles (2.5 m×2.5 m) and personnel (1.2 m×1.2 m).

TABLE 4 Resolving elements, δl, According to Johnson Criterion, forVehicles and Personnel Vehicles Personnel Function (2.5 m × 2.5 m) (1.2m × 1.2 m) Detect (D) (5 Pixels per Target) 50 cm 26 cm Recognize (10Pixels per Target) 25 cm 12 cm Identify (15 Pixels per Target) 16.67cm    8 cm

In various embodiments, different total FOVs may be employed. Forexample, in some system, a 180° FOV is desirable instead of a 360° FOV.The multi-facet mirror systems allow increased sensor resolution in anyFOV. FIG. 17, illustrates a catadioptric geometry for a panoramic 180°Field of View (FOV), with distance, r, and resolving element, δl. InFIG. 17A, the hemispheric FOV is presented, with total number of pixelsequal to 2N_(r) ² and optimized total number of (π/2)N_(r) ². From FIG.17A.

$\begin{matrix}{{\pi\; N_{r}} = {\left. \frac{\pi\; r}{\delta\; l}\Rightarrow N_{r} \right. = \frac{r}{\delta\; l}}} & (13)\end{matrix}$

Comparing FIG. 17A with FIG. 2, δr has been replaced by δl, while,comparing Eq. (13) with Eq. (4), N_(r)-value is twice smaller. This isbecause the Nyquist criterion has been replaced by more practicalJohnson criterion where δr and δl are defined differently (i.e., twopixels per δr, while: 5, 10, or 15 pixels per δl.

The minimum number of superpixels depends on the catadioptric geometry,in the form:4N_(r) ²-Un-optimized spheric (360°) FOV  (14a)πN_(r) ²-Optimized spheric (360°) FOV  (14b)2N_(r) ²-Un-optimized hemispheric (180°) FOV  (14c)(π/2)N_(r) ²-Optimized hemispheric (180°) FOV  (14d)where N_(r)-value is obtained from Eq. (13). Of course, finding any ofthem is sufficient to find all of them by constant factormultiplication.

Tables 5 and 6 summarize the minimum superpixel numbers, by applying Eq.(14b) for optimized spheric FOV, for target detection (D), recognition(R) and identification (I) by using the Johnson criteria for vehiclesand people. The highest number: 9.91·10¹⁰=49.1 Gp, is for personnelidentification for 10 km-distance.

TABLE 5 Minimum Superpixel Numbers of Vehicles (2.5 m × 2.5 m), forVarious Distances, Assuming Optimized Spheric (360°) FOV, for Detection(D), Recognition (R), and Identification (I) Distance (r) 500 m 1 km 2km 5 km 10 km D 3.14 · 10⁶ 1.26 · 10⁷   5 · 10⁷ 3.14 · 10⁸ 1.26 · 10⁹ R1.26 · 10⁷   5 · 10⁷   2 · 10⁸ 1.26 · 10⁹   5 · 10⁹ I  2.8 · 10⁷ 1.13 ·10⁸ 4.52 · 10⁸  2.8 · 10⁹ 1.13 · 10¹⁰

TABLE 6 Minimum Superpixel Numbers of Personnel (1.2 × 1.2 m), forVarious Distances, Assuming Optimized Spheric (360°) FOV, for Detection(D), Recognition (R), and Identification(I) Distance (r) 500 m 1 km 2 km5 km 10 km D 1.36 · 10⁷ 5.45 · 10⁷ 2.18 · 10⁸ 1.36 · 10⁹ 5.49 · 10⁹ R5.45 · 10⁷ 2.18 · 10⁸ 8.72 · 10⁸ 5.45 · 10⁹ 2.18 · 10¹⁰ I 1.23 · 10⁸4.91 · 10⁸ 1.96 · 10⁹ 1.23 · 10¹⁰ 4.91 · 10¹⁰

It should be noted that any value of Tables 5 and 6 is a minimum value,obtained for resolving element size, δl, defining minimum pixel pitch(i.e., distance between two sequent pixels). Therefore, any number oflinear pixels larger than this minimum value also satisfies the Johnsoncriterion for other cases with lower N_(r)-values. Consider, forexample, vehicle recognition (R) at distance 2 km, with πN_(r) ²-valueequal to 2·10⁸, according to Table 5. Assume, that the number ofsuperpixels is 3·10⁸, for example. Then personnel recognition atdistance 1 km, with πN_(r) ²=2.18·10⁸, according to Table 6 alsosatisfies the Johnson criterion. However, vehicle recognition atdistance 5 km, with πN_(r) ²=1.26·10⁹, does not satisfy the Johnsoncriterion, since 3·10⁸ is smaller number than 1.26·10⁹,

In FIG. 17A, the pixels located at the circle with radius N_(r)represent the highest vertical resolution, while pixels located at thecircle with radius smaller than N_(r) represent respectively lowervertical resolution. When the catadioptric is located at zero elevation,for example, typically the highest vertical resolution at zero elevationis assumed, with proportionally lower resolution levels at higherelevations, as in FIG. 17B. When the catadioptric system is located athigher elevation (e.g., located on a tower, or on a hill), however, wehave also negative elevation angles, as illustrated in FIG. 1, FOV 46.In general, azimuthal mirror facets, as discussed with respect to FIG.9, can be always adjusted in order to maximize vertical resolution forany specific elevation required.

On average, the number of vertical pixels required is determined by thefollowing equation:

$\begin{matrix}{{{\Delta\; N_{r}} = {\frac{r \cdot {\Delta\phi}}{\delta\; l} \leq N_{r}}},{{{{if}\mspace{14mu}{\Delta\phi}} \leq 1} = {57.3{^\circ}}}} & (15)\end{matrix}$where Δφ defines a vertical FOV. The vertical FOV, should be smallerthan 57.3°, in order to satisfy, (in average), the pixel geometry ofFIG. 17A. The vertical resolution reduction, is defined by the formula:

$\begin{matrix}{\left( N_{r} \right)_{e} = {{N_{r} - {\Delta\; N_{r}}} = {\frac{r}{\delta\; l}\left( {1 - {\Delta\phi}} \right)}}} & (16)\end{matrix}$For example, for Δφ=30°=π/6=0.52, we obtain

$\begin{matrix}{\left( N_{r} \right)_{e} = {{\frac{r}{\delta\; l}\left( {1 - \frac{\pi}{6}} \right)} = {0.48\mspace{14mu} N_{r}}}} & (17)\end{matrix}$while for Δφ=45°=π/4=0.785, and (N_(r))_(e)=0.215 N_(r); i.e., at thehighest elevation (45°), the vertical resolution is reduced by 78.5%.Therefore, in the case of observation of aerial objects, for example,the vertical resolution should be optimized to respectively higherelevation levels.

In further embodiments, software processing systems may be used toprovide automatic motion detection and tracking capability based onreal-time PRE-AIR (or, RT/PRE-ATR) techniques. Conventional automaticTarget Recognition (ATR) method based on Fourier processing, may bedifficult to implement in some situations because it is not real-timewith current computing technology, and it is not scale and/or rotationinvariant. In contrast, the RT/PRE-ATR is a simplified objectrecognition system that could allow object recognition in real-time andit is scale/rotation-invariant thus, avoiding the ATR problems. It has anumber of automatic steps. First, the embodiment produces a silhouetteor profile of an object (human, animal, car, tank), based on:pixel-by-pixel frame subtraction, object edge extraction (based onnovelty filtering) using such parameters as: mean square error, peaksignal-to-noise ratio (PSNR), profile polar distribution and polarcompliance parameter. Having a number of pixels per object (observed bythe E-ZOOM) and angular resolution per object, δφ, obtained from theformula:

$\begin{matrix}{{({\delta\phi})\left( {\pi\; N_{r}} \right)} = {\left. \pi\Rightarrow{{\delta\phi} \cdot N_{r}} \right. = {\left. 1\Rightarrow{\delta\phi} \right. = \frac{1}{N_{r}}}}} & (18)\end{matrix}$the angular sizes of (usually moving) object may be found by automaticpixel counting. However, from the RT/PRE-ATR, or other patternrecognition method, the object is recognized; i.e., since the type of anobject (car, human, etc.) is known; so, the (real) linear sizes, δa, ofthe object are also known because:

$\begin{matrix}{{\delta\phi} = \frac{\delta\; a}{r}} & (19)\end{matrix}$i.e., by knowing δφ and δa, the r-distance value can be found (it shouldbe emphasized that, since, the operation is not manual, the realdistance of an object is not known).

Assuming that the object is in motion, the real velocity, v, can also beautomatically found from the following formula:

$\begin{matrix}{v = {\frac{\Delta\; s}{\Delta\; t} = {{\omega\; r} = {{\left( \frac{\Delta\phi}{\Delta\; t} \right)r} = {\left( \frac{N_{m} \cdot {\delta\phi}}{\Delta\; t} \right)r}}}}} & (20)\end{matrix}$where: ω-angular velocity, N_(m)—number of pixels passed during theobject motion, Δt—time duration, while δφ and r-values are known. Itshould be noted that r-object distance can be function of time:r=r(t)  (21)where, by knowing relation (21) we can also find the object velocityvector {right arrow over (v)}; i.e., if terrestrial object is movingaway or it is coming closer. For aerial objects (aircraft, helicopter,UAV), there are three {right arrow over (v)}-vector components. Forexample, for N_(r)=8000 (thus, δφ=1.25·10⁻⁴), N_(m)=100, Δt=1 sec, andr=1 km, we obtain v=12.5 m/sec=45 km/h.

The RT/PRE-ATR allows identification of regions of interest (“ROIs”),including both moving and stationary objects, based on edge extraction,within entire panoramic view: 180°×45°. This ROI identification may beused for a variety of purposes, for example, for recording purposes. Inone embodiment the automatically identified ROIs are compressed (e.g.,by MPEG-4), without perceptual information loss, with a firstcompression ratio, such as C₁=200:1, while remaining background iscompressed with a greater compression ratio, such as C₂=2000:1. Then,the average compression ratio, C, is;

$\begin{matrix}{\frac{1}{C} = {\frac{k}{C_{1}} + \frac{\left( {1 - k} \right)}{C_{2}}}} & (22)\end{matrix}$where k is the fraction of the image occupied by ROIs. For example, forC₁=200:1, C₂=2000:1, and k−0.01 (1%), C=1852.

Assuming a total number of CCD superpixels equal to 1 Gp, for example,with 24 bpp (RGB), and frame rate: (FR)=1 fps. Then, un-compressedbandwidth; B_(o)=(10⁹)(24)(1)=24 Gbps, and compressed bandwidth,B=(B_(o)/C)=(24·10⁹)/(1852)=15.8 Mbps. Therefore, the requiredinformation capacity, V, for 48 hours of recording, isV=(15.8·10⁶(48)(3600)=337 GB  (23)

For visualization purposes, assume a display with n_(s)—number ofhorizontal display pixels (vertical display pixels can be calculatedanalogously which is especially important for aerial objects), andequivalent number of CCD superpixels, Ns, where:

$\begin{matrix}{N_{s} = {{\left( \frac{\Delta\alpha}{\pi} \right)\left( {\pi\; N_{r}} \right)} = {({\Delta\alpha})\left( N_{r} \right)}}} & (24)\end{matrix}$where Δα is display screen horizontal FOV, as shown in FIG. 18.

The term “clusterization ratio”, ε, is:

$\begin{matrix}{ɛ = {\frac{N_{s}}{n_{s}} = \frac{({\Delta\alpha}) \cdot N_{r}}{n_{s}}}} & (25)\end{matrix}$where N_(s) is given by Eq. (24), while n_(s)—is given number ofhorizontal display pixels (FIG. 18).

Further embodiments of the invention provide modules configured toprovide de-blurring. In the case of the objects moving with maximumhorizontal velocity, v_(o), and δt—time between two sequent videoframes, where:

$\begin{matrix}{({FR}) = \frac{1}{\delta\; t}} & (26)\end{matrix}$and (FR) is frame rate, the pixel de-blurring condition is calculatedfrom the following equation:

$\begin{matrix}{\frac{{v_{o} \cdot \delta}\; t}{r_{o}} = {\delta\beta}} & (27)\end{matrix}$where r_(o)—minimum object distance and δβ is angular display pixelpitch, related to the CCD superpixel pitch by clusierization ratio:

$\begin{matrix}{{\delta\beta} = {{ɛ \cdot {\delta\phi}} = \frac{ɛ}{N_{r}}}} & (28)\end{matrix}$where ε-coefficient is given by Eq. (25). This relation is because:(δφ)(πN_(r))=π

(δφ)(N_(r))=1. By substituting Eq. (28) into Eq. (27):

$\begin{matrix}{\frac{{v_{o} \cdot \delta}\; t}{r_{o}} = {\frac{ɛ}{N_{r}} = {\frac{\Delta\alpha}{n_{s}} = {\delta\beta}}}} & (29)\end{matrix}$or, by using Eq. (26):

$\begin{matrix}{({FR}) = {{\frac{v_{o}}{r_{o}}\frac{n_{s}}{\Delta\alpha}} = {\frac{v_{o}}{r_{o}}\frac{1}{\delta\beta}}}} & (30)\end{matrix}$where δβ is angular distance between display pixels. Equation (30) isthe basic relation, defining minimum frame rate required to avoidde-blurring. We see that this relation depends only on object kinematic(v_(o), r_(o))—parameters and display parameters: n_(s), Δα. It shouldbe noted that δt-parameter can be also considered as a exposure time. Insuch a case δt-value can be small than that from Eq. (26); i.e.,δt<(FR)⁻¹.

By introducing Eq. (25),

$\begin{matrix}{{({FR}) \cdot ɛ} = \frac{v_{o}N_{r}}{r_{o}}} & (31)\end{matrix}$Accordingly, given kinematic parameters: v_(o), r_(o), and given E-ZOOMparameter, N_(r); then, with (FR) and ε remain unknown until we displayparameters: Δα and n_(s), are defined. Then, both (FR) and ε will befixed. Therefore, in order to avoid deblurring, all three types ofparameters (N_(r)), kinematic (r_(o), v_(o)), and display (Δα, n_(s))should be determined. In typical implementations, the parameters: r_(o),v_(o), N_(r), and n_(s), are given and the software has the flexibilityto adjust:Δα

∈, from Eq. (25)  (32)and, also,∈

(FR), from Eq. (31)  (33)

Thus, the video visualization must be adjusted by software into givenkinematic, display, and catadioptric conditions, defined by thefollowing parameters: v_(o), r_(o), n_(s), and N_(r). In additionN_(r)-number is defined by Eq. (13). This situation is illustrated byTable 7 which is determined by resolving element, δl, and Δα-value, suchas: δl=10 cm and Δα=π=180°, for example. Therefore, starting from r_(o)and Δl, we obtain Nr-value, and for v_(o′)=100 km/h=27.8 m/sec, weobtain (FR)-values, for given, r_(o)-values, by using Eq. (30). Then,from Eq. (31), we obtain ε-values. We see that we need to assume ratherhigh r_(o)-values, in order to obtain relatively low frame rates, whileε-values are still high, even for large Δα=π=180°.

TABLE 7 Frame Rate (FR) and ε-Values for Various Minimum Distances,r_(o), Assuming: v_(o) = 100 km/h, Δα = π = 180°, δl = 10 cm, n_(s) =1000 r_(o) 500 m 1 km 2 km 5 km 10 km N_(r) 5000 10,000 20,000 50,000100,000 (FR) 16.4 fps 8.2 fps 4.1 fps 1.77 fps 0.88 fps ε 15.7 31.4 62.8157 314

The visualization structure is defined by the following technologyelements; where their representative eight (8) parameters are alsogiven:

-   -   1) Software (∈)    -   2) Visualization (n_(s), Δα)    -   3) E-ZOOM Sensor (N_(r))    -   4) Kinematics (v_(o), r_(o))    -   5) Resolution (δl)    -   6) Video (FR)

The object kinemetrics (v_(o), r_(o)) and object resolution (δl) definesensor (N_(r)) and, as a result, Equation (31) is given, with definedproduct: (FR)·∈. Then, by adding visualization parameter: δβ=Δα/n_(s),Eqs. (25) and (31) are obtained, defining parameters: (FR) and ∈,separately.

As used herein, the term catoptric refers to reflection optics, the termdioptric refers to lens and pinhole optics, and the term catadioptricrefers to combinations of reflection and lens or pinhole optics. As usedherein, the term module might describe a given unit of functionalitythat can be performed in accordance with one or more embodiments of thepresent invention. As used herein, a module might be implementedutilizing any form of hardware, software, or combination thereof. Forexample, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs,FPGAs, logical components, software routines or other mechanisms mightbe implemented to make up a module. In implementation, the variousmodules described herein might be implemented as discrete modules or thefunctions and features described can be shared in part or in total amongone or more modules. In other words, as would be apparent to one ofordinary skill in the art after reading this description, the variousfeatures and functional described herein may be implemented in any givenapplication and can be implemented in one or more separate or sharedmodules in various combinations and permutations. Even though variousfeatures or elements of functionality may be individually described orclaimed as separate modules, one of ordinary skill in the art willunderstand that these features and functionality can be shared among oneor more common software and hardware elements, and such descriptionshall not require or imply that separate hardware or software componentsare used to implement such features or functionality.

Where components or modules of the invention are implemented in whole orin part using software, in one embodiment, these software elements canbe implemented to operate with a computing or processing module capableof carrying out the functionality described with respect thereto. Onesuch example computing module is shown in FIG. 19. Various embodimentsare described in terms of this example-computing module 600. Afterreading this description, it will become apparent to a person skilled inthe relevant art how to implement the invention using other computingmodules or architectures.

Referring now to FIG. 19, computing module 600 may represent, forexample, computing or processing capabilities found within desktop,laptop and notebook computers; hand-held computing devices (PDA's, smartphones, cell phones, palmtops, etc.); mainframes, supercomputers,workstations or servers; or any other type of special-purpose orgeneral-purpose computing devices as may be desirable or appropriate fora given application or environment. Computing module 600 might alsorepresent computing capabilities embedded within or otherwise availableto a given device. For example, a computing module might be found inother electronic devices such as, for example, digital camerasnavigation systems, cellular telephones, portable computing devices,modems, routers, WAPs, terminals and other electronic devices that mightinclude some form of processing capability.

Computing module 600 might include, for example, one or more processors,controllers, control modules, or other processing devices, such as aprocessor 604. Processor 604 might be implemented using ageneral-purpose or special-purpose processing engine such as, forexample, a microprocessor, controller, or other control logic. In theillustrated example, processor 604 is connected to a bus 602, althoughany communication medium can be used to facilitate interaction withother components of computing module 600 or to communicate externally.

Computing module 600 might also include one or more memory modules,simply referred to herein as main memory 608. For example, preferablyrandom access memory (RAM) or other dynamic memory, might be used forstoring information and instructions to be executed by processor 604.Main memory 608 might also be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 604. Computing module 600 might likewise include aread only memory (“ROM”) or other static storage device coupled to bus602 for storing static information and instructions for processor 604.

The computing module 600 might also include one or more various forms ofinformation storage mechanism 610, which might include, for example, amedia drive 612 and a storage unit interface 620. The media drive 612might include a drive or other mechanism to support fixed or removablestorage media 614. For example, a hard disk drive, a floppy disk drive,a magnetic tape drive, an optical disk drive, a CD or DVD drive (R orRW), or other removable or fixed media drive might be provided.Accordingly, storage media 614 might include, for example, a hard disk,a floppy disk, magnetic tape, cartridge, optical disk, a CD or DVD, orother fixed or removable medium that is read by, written to or accessedby media drive 612. As these examples illustrate, the storage media 614can include a computer usable storage medium having stored thereincomputer software or data.

In alternative embodiments, information storage mechanism 610 mightinclude other similar instrumentalities for allowing computer programsor other instructions or data to be loaded into computing module 600.Such instrumentalities might include, for example, a fixed or removablestorage unit 622 and an interface 620. Examples of such storage units622 and interfaces 620 can include a program cartridge and cartridgeinterface, a removable memory (for example, a flash memory or otherremovable memory module) and memory slot, a PCMCIA slot and card, andother fixed or removable storage units 622 and interfaces 620 that allowsoftware and data to be transferred from the storage unit 622 tocomputing module 600.

Computing module 600 might also include a communications interface 624.Communications interface 624 might be used to allow software and data tobe transferred between computing module 600 and external devices.Examples of communications interface 624 might include a modem orsoftmodem, a network interface (such as an Ethernet, network interfacecard, WiMedia, IEEE 802.XX or other interface), a communications port(such as for example, a USB port, IR port, RS232 port Bluetooth®interface, or other port), or other communications interface. Softwareand data transferred via communications interface 624 might typically becarried on signals, which can be electronic, electromagnetic (whichincludes optical) or other signals capable of being exchanged by a givencommunications interface 624. These signals might be provided tocommunications interface 624 via a channel 628. This channel 628 mightcarry signals and might be implemented using a wired or wirelesscommunication medium. Some examples of a channel might include a phoneline, a cellular link, an RF link, an optical link, a network interface,a local or wide area network, and other wired or wireless communicationschannels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to media such as, forexample, memory 608, storage unit 620, media 614, and channel 628. Theseand other various forms of computer program media or computer usablemedia may be involved in carrying one or more sequences of one or moreinstructions to a processing device for execution. Such instructionsembodied on the medium, are generally referred to as “computer programcode” or a “computer program product” (which may be grouped in the formof computer programs or other groupings). When executed suchinstructions might enable the computing module 600 to perform featuresor functions of the present invention as discussed herein.

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample only, and not of limitation. Likewise, the various diagrams maydepict an example architectural or other configuration for theinvention, which is done to aid in understanding the features andfunctionality that can be included in the invention. The invention isnot restricted to the illustrated example architectures orconfigurations, but the desired features can be implemented using avariety of alternative architectures and configurations. Indeed, it willbe apparent to one of skill in the art how alternative functional,logical or physical partitioning and configurations can be implementedto implement the desired features of the present invention. Also amultitude of different constituent module names other than thosedepicted herein can be applied to the various partitions. Additionally,with regard to flow diagrams, operational descriptions and methodclaims, the order in which the steps are presented herein shall notmandate that various embodiments be implemented to perform the recitedfunctionality the same order unless the context dictates otherwise.

Although the invention is described above in terms of various exemplaryembodiments and implementations, it should be understood that thevarious features, aspects and functionality described in one or more ofthe individual embodiments are not limited in their applicability to theparticular embodiment with which they are described, but instead can beapplied, alone or in various combinations, to one or more of the otherembodiments of the invention, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentinvention should not be limited by any of the above-described exemplaryembodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof, the terms “a” or“an” should be read as meaning “at least one” “one or more” or the like;and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, can be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

The invention claimed is:
 1. An apparatus, comprising: a catoptricstructure configured to reflect light from a first field of view onto animage plane comprising a plurality of sensors, the catoptric structurehaving a surface comprising a plurality of facets separated by aplurality of catoptric regions; wherein a facet of the plurality of thefacets has a second field of view that is smaller than the first fieldof view; wherein the plurality of sensors comprises a plurality ofseparate pixel arrays, and each facet as a focal point corresponding toa predetermined point on a corresponding of one of the plurality ofpixel arrays; and wherein a catoptric region of the plurality ofcatoptric regions between two facets of the plurality of facets isconfigured to reflect light from a world point within the first field ofview onto two separate pixel arrays.
 2. The apparatus of claim 1,wherein the catoptric region between the two facets is configured toreflect light from the world point within the field of view ontoneighboring edge regions of the pixel arrays.
 3. The apparatus of claim2, wherein each facet of the plurality of facets has an effectiveviewpoint, and wherein each image of the viewpoints of the plurality offacets are imaged onto a separate pixel array.
 4. The apparatus of claim2. wherein each facet of the plurality of facets has an effectiveviewpoint, and wherein the effective viewpoints of the plurality offacets coincide at a focal plane of the catoptric structure.
 5. Theapparatus of claim 2, wherein each facet of the plurality of facets hasan effective viewpoint, and wherein the effective viewpoints of theplurality of facets do not coincide at a focal plane of the catoptricstructure.
 6. The apparatus of claim 2, wherein: the first field of viewhas a first horizontal component and a first vertical component; thesecond field of view has a second horizontal component that issubstantially the same as the first horizontal component; and the secondfield of view has a second vertical component that is smaller than thefirst vertical component.
 7. The apparatus of claim 6, wherein a facetof the plurality of facets has a surface profile comprising a portion ofa surface of revolution of a skewed or unskewed conic section from afirst angle to a second angle.
 8. The apparatus of claim 7, wherein theangular distance between the first angle and the second angle determinesthe width of edge regions of the two sensors.
 9. The apparatus of claim7, wherein the angular distance between the first angle and the secondangle is $\frac{{FOV}_{H}}{n}$ plus between 1 and 15 degrees based onthe chip boundary thickness, where FOV_(H) is the first horizontalcomponent and n is the number of facets of the plurality of facets. 10.The apparatus of claim 7, wherein the conic section is a hyperbola. 11.The apparatus of claim 2, wherein: the first field of view has a firsthorizontal component and a first vertical component; the second field ofview has a second horizontal component that is smaller than the firsthorizontal component; and the second field of view has a second verticalcomponent that is smaller than the first vertical component.
 12. Acatadioptric imaging system, comprising: a catoptric structureconfigured to reflect light from a first field of view onto an imageplane comprising a plurality of sensors, the catoptric structure havinga surface comprising a plurality of facets separated by a plurality ofcatoptric regions, wherein a facet of the plurality of the facets has asecond field of view that is smaller than the first field of view;wherein the plurality of sensors comprises a plurality of separate pixelarrays, and each facet has a focal point corresponding to apredetermined point on a corresponding of one of the plurality of pixelarrays; and wherein a catoptric region of the plurality of catoptricregions between two facets of the plurality of facets is configured toreflect light from a world point within the first field of view onto twoof the separate pixel arrays.
 13. The catadioptric imaging system ofclaim 12, wherein the catoptric region between the two facets isconfigured to reflect light from the world point within the field ofview onto neighboring edge regions of the pixel arrays.
 14. Thecatadioptric imaging system of claim 13, wherein each facet of theplurality of facets has an effective viewpoint, and wherein each imageof the viewpoints of the plurality of facets are imaged onto a separatepixel array.
 15. The catadioptric imaging system of claim 14, whereinthe lens has a multi-viewpoint topology corresponding the viewpoints ofthe plurality of facets.
 16. The catadioptric imaging system of claim13, wherein each facet of the plurality of facets has an effectiveviewpoint, and wherein the effective viewpoints of the plurality offacets coincide at a focal plane of the catoptric structure.
 17. Thecatadioptric imaging system of claim 13, wherein each facet of theplurality of facets has an effective viewpoint, and wherein theeffective viewpoints of the plurality of facets do not coincide at afocal plane of the catoptric structure.
 18. The catadioptric agingsystem of claim 13, wherein: the first field of view has a firsthorizontal component and a first vertical component; the second field ofview has a second horizontal component that is substantially the same asthe first horizontal component; and the second field of view has asecond vertical component that is smaller than the first verticalcomponent.
 19. The catadioptric imaging system of claim 18, wherein thearray of sensors comprises an array of rectangular sensors arranged in aradial array.
 20. The catadioptric imaging system of claim 18, wherein afacet of the plurality of facets has a surface profile comprising aportion of a surface of revolution of a skewed or unskewed conic sectionfrom a first angle to a second angle.
 21. The catadioptric imagingsystem of claim 20, wherein the angular distance between the first angleand the second angle determines the width of edge regions of the twosensors.
 22. The catadioptric imaging system of claim 20, wherein theangular distance between the first angle and the second angle is$\frac{{FOV}_{H}}{n}$ plus between 1 and 15 degrees based on the chipboundary thickness, where FOV_(H) is the first horizontal component andn is the number of facets of the plurality of facets.
 23. Thecatadioptric imaging system of claim 20, wherein the conic section is ahyperbola.
 24. The catadioptric imaging system of claim 13, wherein: thefirst field of view has a first horizontal component and a firstvertical component; the second field of view has a second horizontalcomponent that is smaller than the first horizontal component; and thesecond field of view has a second vertical component that is smallerthan the first vertical component.
 25. The catadioptric imaging systemof claim 24, wherein the facets of the plurality of facets are arrangedon the surface of the catoptric structure in a plurality of azimuthalzones.
 26. The catadioptric imaging system of claim 25, wherein theplurality of sensors is arrayed into a plurality of different resolutionzones corresponding to the plurality of azimuthal zones.
 27. Theapparatus of claim 1, wherein the predetermined field of view is thefirst field of view.
 28. The catadioptric imaging system of claim 12,wherein the predetermined field of view is the first field of view. 29.The apparatus of claim 1, wherein the light reflected from a world pointwithin the first field of view onto two separate pixel arrays createsredundant images on at least a portion of each of the two separate pixelarrays.
 30. The apparatus of claim 29, wherein the two separate pixelarrays are adjacent, and the redundant images on at least a portion ofeach of the two separate pixel arrays are used to stitch together imagesfrom each pixel array, effectively eliminating a gap between the twoadjacent pixel arrays.
 31. The apparatus of claim 1, wherein theplurality of sensors comprises a separate sensor corresponding to eachfacet of the plurality of facets.
 32. The apparatus of claim 1, whereineach facet comprises a section of a paraboloid.
 33. The apparatus ofclaim 1, wherein the plurality of facets of the catopric structure arearranged horizontally and azimuthally about the structure.
 34. Theapparatus of claim 33, wherein the plurality of sensors comprises aseparate sensor corresponding to each facet of the plurality of facets.35. The apparatus of claim 34, wherein the facets are arranged in aplurality of zones, and further wherein the plurality sensors comprisedifferent resolution sensors corresponding to each of the plurality ofzones.
 36. The apparatus of claim 12, wherein the light reflected from aworld point within the first field of view onto two separate pixelarrays creates redundant images on at least a portion of each of the twoseparate pixel arrays.
 37. The apparatus of claim 36, wherein the twoseparate pixel arrays are adjacent, and the redundant images on at leasta portion of each of the two separate pixel arrays are used to stitchtogether images from each pixel array, effectively eliminating a gapbetween the two adjacent pixel arrays.
 38. The apparatus of claim 12,wherein the plurality of sensors comprises a separate sensorcorresponding to each facet of the plurality of facets.
 39. Theapparatus of claim 12, wherein each facet comprises a section of aparaboloid.
 40. The apparatus of claim 12, wherein the plurality offacets of the catopric structure are arranged horizontally andazimuthally about the structure.
 41. The apparatus of claim 40, whereinthe plurality of sensors comprises a separate sensor corresponding toeach facet of the plurality of facets.
 42. The apparatus of claim 41,wherein the facets are arranged in a plurality of zones, and furtherwherein the plurality sensors comprise different resolution sensorscorresponding to each of the plurality of zones.